Emotions during learning: The first steps toward an affect sensitive intelligent tutoring system

In an attempt to discover links between learning and emotions, this study adopted an emote-aloud procedure where participants were recorded as they verbalized their affective states while interacting with an intelligent tutoring system, AutoTutor. Participants’ facial expressions were coding using the Facial Action Coding System and analyzed using association rule mining techniques. The resulting rules are discussed along with implications to the larger project of improving the AutoTutor system into a nonintrusive affect sensitive intelligent tutoring system. While the 20 century has been ripe with learning theory, these theories have mostly ignored the importance of the link between a persons emotions or affective states and learning (Meyer, & Turner, 2002). However, toward the end of the twentieth century, emotions started to get more attention. Some seminal contributions to the literature include the facial action coding system by Ekman & Friesen (1978), Stein and Levine’s (1991) theory of goals and emotion, Cognitive theory of Emotion (Ortony, Clore, & Collins, 1988), and Russell’s (2003) theory of emotion. Ekman and Friesen (1978) highlighted the expressive aspects of emotions with their Facial Action Coding System that allowed for “basic emotions” to be identified by coding specific facial behaviors based on the muscles that produce them. Each movement in the face is referred to as an action unit. There are approximately 58 action units. These prototypical facial patterns were used to identify the emotions of happiness, sadness, surprise, disgust, anger, and fear (Ekman & Friesen, 1978; Elfenbein & Ambady, 2002). The coding system was tested primarily on static pictures rather than on changing expressions over time. Unfortunately, for those researchers interesting in the role of emotions in learning, it is doubtful whether these 6 emotions are frequent and functionally significant in the learning process (Kapoor,Mota, & Picard, 2001). More generally, some researchers have challenged the adequacy of basing a complete theory of emotions on these “basic” emotions (Rozin & Cohen, 2003). The claim has been made that cognition, motivation, and emotions are the three components of learning (Snow, Corno, & Jackson, 1996). Until recently, emotion has been viewed as source of motivational energy (Harter, 1981; Miserandino, 1996; Stipek, 1998), but they have not focused on as independent factor in learning or motivation (Ford, 1992; Meyer & Turner, 2002). However, in the last decade, the link between emotions and learning has received more attention (e.g. Craig, Graesser, Sullins, & Gholson, in press; Kort, Reilly, & Picard, 2001; Picard 1997; Meyer & turner, 2002). Kort, Reilly, and Picard (2001) put forth a four quadrant model that explicitly links learning and affective states. The learning process is broken up by two axes, vertical and horizontal, labeled learning and affect respectively. The learning axis ranges from “constructive learning” at the top, where new information is being integrated into schemas, and “un-learning” at the bottom where misconceptions are identified and removed from schemas. The affect axis ranges from positive affect on the right to negative affect on the left. According to this model, learners move around the circle from a state of ease, to encountering misconceptions, to discarding misconceptions, to new understanding, and then back into a state of ease. For a more detailed description of this model see Kort et al (2001) or Craig et al. (in press). Much of the current research into the link between emotions (or affective states) and learning has come from the area of user modeling. Much work in this field has focused on identifying the users’ emotions as they interact with computer systems such as tutoring systems (Fan, Sarrafzadeh, Overmyer, Hosseini, Biglari-Abhari, & Bigdeli, 2003) or educational games (Conati, 2002). However, many of the types of systems only assess intensity, or valence (Ball & Breeze, 2000), or a single affective state (Hudlicka & McNeese, 2002). For example, Guhe, Gray, Schoelles, and Ji (2004) have recently created a system in which a user is monitored in an attempt to detect confusion during interaction with an intelligent tutoring system. The problem with this method is that one affective state is not sufficient to encompass the whole gamut of learning (Conati, 2002). Recently, Craig, et al. (in press) for example, presented evidence that boredom, confusion, and flow were all correlated with learning gains. Another problem with the single state detection approach is that the person’s reaction to the presented material can change depending on their goals, preferences, expectations and knowledge state (Conati, 2002). The current research reports the first step in a larger project to integrate affect sensing into an intelligent tutoring system, AutoTutor (Graesser, K. Wiemer-Hastings, P. Wiemer-Hastings, Harter, Kreuz, & TRG, 1999; Graesser, Person, Harter, & TRG, 2001). The purpose of this study is two fold. First, we want to identify affective states that occur frequently during learning. The affective states of interest in this study were anger, boredom, confusion, contempt, curiosity, disgust, eureka, and frustration. Second, we have adopted the Facial Action Coding System (Ekman & Friesen, 1978) as a method to identify these affective states. Association rule mining techniques were employed to locate frequent sets of action units that occur together and to extract association rules that could conditionally influence the presence of action units on the face. Association rule mining (Agarwal, Imielinski, & Swami, 1993) is a widely used technique to find interesting associations among sets of data items. Association rules are probabilistic in nature and take the form “Antecedent → Consequent [support, confidence]”. The antecedent is an item or a set of items whose occurrence influences the occurrence of the consequent (also an item or a set of items). The support of a rule measures its usefulness and is the probability that a record in the data set will contain both the antecedent and the consequent. The confidence measures its certainty and is the conditional probability that a record containing the antecedent will contain the consequent. It provides a measure of the influence the antecedent has on the presence of the consequent. The a prori algorithm (Agarwal and Srikant, 1994) was first used to mine sets of frequent action units (called frequent itemsets). Association rules were then obtained from the frequent itemsets mined.